IES PRACTICE GUIDE 



WHAT WORKS CLEARINGHOUSE 




NCEE 2009-4067 ■ W W and REGIONAL ASSISTANCE 

U.S. DEPARTMENT OF EDUCATION Institute of Education Sciences 





The Institute of Education Sciences (IES) publishes practice guides in education 

to bring the best available evidence and expertise to bear on the types of challenges 
that cannot currently be addressed by a single intervention or program. Authors of 
practice guides seldom conduct the types of systematic literature searches that are 
the backbone of a meta-analysis, although they take advantage of such work when 
it is already published. Instead, authors use their expertise to identify the most im- 
portant research with respect to their recommendations and conduct a search of 
recent publications to ensure that the research supporting the recommendations 
is up-to-date. 

Unique to IES-sponsored practice guides is that they are subjected to rigorous exter- 
nal peer review through the same office that is responsible for independent reviews 
of other IES publications. A critical task for peer reviewers of a practice guide is to 
determine whether the evidence cited in support of particular recommendations is 
up-to-date and that studies of similar or better quality that point in a different di- 
rection have not been ignored. Because practice guides depend on the expertise of 
their authors and their group decision making, the content of a practice guide is not 
and should not be viewed as a set of recommendations that in every case depends 
on and flows inevitably from scientific research. 

The goal of this practice guide is to formulate specific and coherent evidence-based 
recommendations for use by educators and education administrators to create the 
organizational conditions necessary to make decisions using student achievement 
data in classrooms, schools, and districts. The guide provides practical, clear in- 
formation on critical topics related to data-based decision making and is based on 
the best available evidence as judged by the panel. Recommendations presented in 
this guide should not be construed to imply that no further research is warranted 
on the effectiveness of particular strategies for data-based decision making. 
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Introduction 

As educators face increasing pressure 
from federal, state, and local accountabil- 
ity policies to improve student achieve- 
ment, the use of data has become more 
central to how many educators evaluate 
their practices and monitor students’ aca- 
demic progress. 1 Despite this trend, ques- 
tions about how educators should use data 
to make instructional decisions remain 
mostly unanswered. In response, this 
guide provides a framework for using stu- 
dent achievement data to support instruc- 
tional decision making. These decisions 
include, but are not limited to, how to 
adapt lessons or assignments in response 
to students’ needs, alter classroom goals 
or objectives, or modify student-grouping 
arrangements. The guide also provides 
recommendations for creating the orga- 
nizational and technological conditions 
that foster effective data use. Each rec- 
ommendation describes action steps for 
implementation, as well as suggestions 
for addressing obstacles that may impede 
progress. In adopting this framework, edu- 
cators will be best served by implement- 
ing the recommendations in this guide 
together rather than individually. 

The recommendations reflect both the ex- 
pertise of the panelists and the findings 
from several types of studies, including 
studies that use causal designs to examine 
the effectiveness of data use interventions, 
case studies of schools and districts that 
have made data-use a priority, and obser- 
vations from other experts in the field. The 
research base for this guide was identi- 
fied through a comprehensive search for 
studies evaluating academically oriented 
data-based decision-making interventions 
and practices. An initial search for litera- 
ture related to data use to support instruc- 
tional decision making in the past 20 years 
yielded more than 490 citations. Of these, 
64 used experimental, quasi-experimental, 



1. Knapp et al. (2006). 



and single subject designs to examine 
whether data use leads to increases in 
student achievement. Among the studies 
ultimately relevant to the panel’s recom- 
mendations, only six meet the causal va- 
lidity standards of the What Works Clear- 
inghouse (WWC) and were related to the 
panel’s recommendations. 2 

To indicate the strength of evidence sup- 
porting each recommendation, the panel 
relied on the WWC standards for determin- 
ing levels of evidence, described below and 
in Table 1. It is important for the reader to 
remember that the level of evidence rating 
is not a judgment by the panel on how ef- 
fective each of these recommended prac- 
tices will be when implemented, nor is it 
a judgment of what prior research has to 
say about the effectiveness of these prac- 
tices. The level of evidence ratings reflect 
the panel’s judgment of the validity of 
the existing literature to support a causal 
claim that when these practices have been 
implemented in the past, positive effects 
on student academic outcomes were ob- 
served. They do not reflect judgments of 
the relative strength of these positive ef- 
fects or the relative importance of the in- 
dividual recommendations. 

A strong rating refers to consistent and 
generalizable evidence that an inter- 
vention strategy or program improves 
outcomes. 3 

A moderate rating refers either to evidence 
from studies that allow strong causal con- 
clusions but cannot be generalized with 
assurance to the population on which a 
recommendation is focused (perhaps be- 
cause the findings have not been widely 



2. Reviews of studies for this practice guide ap- 
plied WWC Version 1.0 standards. See Version 1.0 
standards at http://ies.ed.gov/ncee/wwc/pdf/ 
wwc_versionl_standards.pdf. 

3. Following WWC guidelines, improved out- 
comes are indicated by either a positive, statisti- 
cally significant effect or a positive, substantively 
important effect size (i.e., greater than 0.25). 
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replicated) or to evidence from studies that 
are generalizable but have more causal 
ambiguity than that offered by experi- 
mental designs (e.g., statistical models of 
correlational data or group comparison de- 
signs for which equivalence of the groups 
at pretest is uncertain). 

A low rating refers to evidence either from 
studies such as case studies and descrip- 
tive studies that do not meet the stan- 
dards for moderate or strong evidence or 
from expert opinion based on reasonable 
extrapolations from research and theory. 
A low level of evidence rating indicates 
that the panel did not identify a body of 
research demonstrating effects of imple- 
menting the recommended practice on 
student achievement. The lack of a body of 
valid evidence may simply mean that the 
recommended practices are not feasible or 
are difficult to study in a rigorous, experi- 
mental fashion . 4 In other cases, it means 



4. For more information, see the WWC Frequently 
Asked Questions page for practice guides, http:// 
ies.ed.gov/ncee/wwc/references/idocviewer/ 
doc.aspx?docid=15&tocid=3. 



that researchers have not yet studied a 
practice or that there is weak or conflicting 
evidence of effectiveness. Policy interest in 
topics of current study thus can arise be- 
fore a research base has accumulated on 
which recommendations can be based. 

Under these circumstances, the panel ex- 
amined the research it identified on the 
topic and combined findings from that 
research with its professional expertise 
and judgments to arrive at recommenda- 
tions. However, that a recommendation 
has a low level of evidence should not be 
interpreted as indicating that the panel 
believes the recommendation is unimport- 
ant. The panel has decided that all five rec- 
ommendations are important and, in fact, 
encourages educators to implement all of 
them to the extent that state and district 
resources and capacity allow. 
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Table 1. Institute of Education Sciences levels of evidence for practice guides 



Strong 


In general, characterization of the evidence for a recommendation as strong requires both 
studies with high internal validity (i.e., studies whose designs can support causal conclu- 
sions) and studies with high external validity (i.e., studies that in total include enough of 
the range of participants and settings on which the recommendation is focused to sup- 
port the conclusion that the results can be generalized to those participants and settings). 
Strong evidence for this practice guide is operationalized as 

• A systematic review of research that generally meets WWC standards (see http://ies. 
ed.gov/ncee/wwc/) and supports the effectiveness of a program, practice, or approach 
with no contradictory evidence of similar quality; OR 

• Several well-designed, randomized controlled trials or well-designed quasi-experi- 
ments that generally meet WWC standards and support the effectiveness of a program, 
practice, or approach, with no contradictory evidence of similar quality; OR 

• One large, well-designed, randomized controlled, multisite trial that meets WWC stan- 
dards and supports the effectiveness of a program, practice, or approach, with no 
contradictory evidence of similar quality; OR 

• For assessments, evidence of reliability and validity that meets the Standards for 
Educational and Psychological Testing. 3 


Moderate 


In general, characterization of the evidence for a recommendation as moderate requires 
studies with high internal validity but moderate external validity or studies with high 
external validity but moderate internal validity. In other words, moderate evidence is 
derived from studies that support strong causal conclusions but generalization is uncer- 
tain or studies that support the generality of a relationship but the causality is uncertain. 
Moderate evidence for this practice guide is operationalized as 

• Experiments or quasi-experiments generally meeting WWC standards and supporting 
the effectiveness of a program, practice, or approach with small sample sizes and/ 
or other conditions of implementation or analysis that limit generalizability and no 
contrary evidence; OR 

• Comparison group studies that do not demonstrate equivalence of groups at pretest 
and, therefore, do not meet WWC standards but that (1) consistently show enhanced 
outcomes for participants experiencing a particular program, practice, or approach 
and (2) have no major flaws related to internal validity other than lack of demonstrated 
equivalence at pretest (e.g., only one teacher or one class per condition, unequal 
amounts of instructional time, highly biased outcome measures); OR 

• Correlational research with strong statistical controls for selection bias and for dis- 
cerning influence of endogenous factors and no contrary evidence; OR 

• For assessments, evidence of reliability that meets the Standards for Educational and 
Psychological Testing 13 but with evidence of validity from samples not adequately rep- 
resentative of the population on which the recommendation is focused. 


Low 


In general, characterization of the evidence for a recommendation as low means that the 
recommendation is based on expert opinion derived from strong findings or theories in 
related areas and/or expert opinion buttressed by direct evidence that does not rise to 
the moderate or strong level. Low evidence is operationalized as evidence not meeting 
the standards for the moderate or strong level. 



a. American Educational Research Association, American Psychological Association, and National Council on 
Measurement in Education (1999). 

b. Ibid. 
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The What Works Clearinghouse 
standards and their relevance to 
this guide 

In terms of the levels of evidence indi- 
cated in Table 1, the panel relied on WWC 
evidence standards to assess the quality 
of evidence supporting educational pro- 
grams and practices. The WWC evaluates 
evidence for the causal validity of instruc- 
tional programs and practices according 
to WWC standards. Information about 
these standards is available at http://ies. 
ed.gov/ncee/wwc/pdf/wwc_versionl_ 
standards.pdf. The technical quality of 
each study is rated and placed into one of 
three categories: 

• Meets Evidence Standards for random- 
ized controlled trials and regression 
discontinuity studies that provide the 
strongest evidence of causal validity. 

• Meets Evidence Standards with Res- 
ervations for all quasi-experimental 
studies with no design flaws and ran- 
domized controlled trials that have 
problems with randomization, attri- 
tion, or disruption. 

• Does Not Meet Evidence Screens for 
studies that do not provide strong evi- 
dence of causal validity. 



Following the recommendations and sug- 
gestions for carrying out the recommen- 
dations, Appendix D presents more in- 
formation on the research evidence that 
supports each recommendation. 
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